[STACKED on #1431] libext: ext4 sequential-read performance (read-ahead + async prefetch) - #1447
Draft
gburd wants to merge 9 commits into
Draft
[STACKED on #1431] libext: ext4 sequential-read performance (read-ahead + async prefetch)#1447gburd wants to merge 9 commits into
gburd wants to merge 9 commits into
Conversation
Two gaps in the ext (lwext4) filesystem module for the Postgres-over-ext goal. 1. fsync durability. ext mounts with lwext4 block-cache write-back enabled, so a write only reaches the disk when the block cache is flushed -- but vop_fsync was vop_nullop, so fsync(2)/fdatasync(2) on an ext file persisted nothing. Implement ext_fsync() to flush the device's block cache (like ext_sync does at unmount), making written data durable. The cache is shared per device, so this persists the file along with any other dirty buffers, which is correct if slightly more than the theoretical per-inode minimum. 2. Page-cache bridge (vop_cache). The vop_cache slot was null, so mmap faults on ext files went through the block layer on every fault, unlike ROFS and ZFS which populate the shared page cache. Add ext_map_cached_page(): on a VOP_CACHE call it reads one page-aligned page of file data into a freshly allocated page and hands it to pagecache::map_read_cached_page(), warming the read cache so subsequent faults and readahead are served from it. This is an allocate-and-copy bridge (lwext4's block-cache buffers are not page-aligned/shareable the way ROFS's read-around cache is); a zero-copy borrow-and-pin bridge like the ZFS ARC one can follow if it shows up hot. Because the module is built with -fno-rtti and <osv/pagecache.hh> pulls in <osv/trace.hh> (which uses typeid), the two page-cache symbols we need are declared minimally instead of including the heavy header (the uio carries the hashkey opaquely, so its layout is never needed). Add tests/tst-ext4-rw.cc: mmap a pre-populated ext4 file and verify the pattern survives the vop_cache bridge (first fault + cached re-read), then write a file, fsync it, and read it back (plus fdatasync and a fresh re-open). Verified on OSv under KVM with an ext4 second disk (created with mkfs.ext4 -b 4096 -O ^64bit,^metadata_csum, which lwext4 supports). Known pre-existing limitation (not introduced here, out of scope for this PR): libext's inode-delete path does not set the inode dtime, so Linux e2fsck flags "deleted inode has zero dtime" on a disk after OSv deletes a file. A fresh disk that OSv only reads/writes+fsyncs (no delete) fscks clean. Tracked as a follow-up in the ext write-path correctness work.
Follow-up to the fsync/page-cache work: libext's inode-delete path freed the inode from the bitmap but never set its on-disk deletion time (dtime), so after OSv created and deleted a file, Linux e2fsck flagged "Deleted inode NN has zero dtime" and reported the filesystem as still having errors. lwext4's own delete path marks the inode with ext4_inode_set_del_time(inode, -1L) before ext4_fs_free_inode(), but that symbol is not exported from liblwext4.so. Add a small ext_mark_inode_deleted() helper that sets inode->deletion_time = 0xffffffff directly (byte-order invariant, so no to_le32() needed) and call it before each ext4_fs_free_inode() in the module (unlink, rmdir, delete-on-last-close, delete-outstanding-on-unmount, and the dir_link allocation-rollback path). Verified: after OSv boots an ext4 second disk, mmaps/reads a file, writes and fsyncs another, then deletes it, `e2fsck -n -f` on the disk now exits 0 (clean), where before it reported the zero-dtime error. tst-ext4-rw still passes.
A filesystem-agnostic micro-benchmark (tst-ext4-bench) measuring sequential write+fsync, sequential read, and 4K random read throughput in MB/s, so the ext4/libext path can be compared A/B against raw virtio-blk and against Linux ext4 on the same storage. Point its argument at any mount (ext, zfs, rofs) for a cross-filesystem comparison.
ext_read allocated a full-size aligned bounce buffer, had lwext4 read into it, then uiomove()'d it to the caller - a per-read malloc plus a second full-size memcpy on every read; ext_write mirrored this in reverse. On the common path (a single contiguous iovec) hand lwext4 the caller's buffer directly. This removes the allocation and one memcpy per read/write and fixes a latent free()/free_contiguous_aligned() mismatch in both error paths. It is a correctness/cleanup change: an A/B on local NVMe showed throughput unchanged (the memcpy is not the bottleneck), which confirms the sequential-read gap vs Linux is the absence of async read-ahead, not copy overhead - that is a separate, larger change (see the ext4 perf notes).
The local-NVMe A/B showed ext4 sequential reads at ~33% of Linux while writes were at parity and random reads ~86%. The gap is that ext_read issues one synchronous bio per read() with no prefetch (ext4_blocks_get_direct bypasses lwext4's block cache), so a stream of small sequential reads cannot keep the device pipeline busy - larger reads were measurably faster purely from amortizing the per-read() round-trip. Add a per-vnode sequential read-ahead cache (in ext_vdata): on a detected sequential single-iovec read smaller than the window, fill a 1 MiB window from disk once and serve that read and subsequent in-window reads from it (a memcpy, no I/O). This turns N small synchronous reads into 1 large read + N copies. The window is invalidated on any write to the file and freed when the vnode goes inactive; access is serialized by a per-vnode mutex. tst-ext4-bench now writes an absolute-offset pattern and verifies every byte on the sequential read, so the benchmark doubles as a read-ahead correctness test (no VERIFY FAIL = the cache returns correct data across window boundaries).
The synchronous 1 MiB read-ahead (6a7990d) reads a window, lets the app consume it, then reads the next window - so the NVMe queue goes idle during the app's compute. Linux keeps the device busy via async prefetch. Add double buffering: two 1 MiB windows (cur + next) and a per-vnode worker pthread. While the app consumes cur (served by memcpy), the worker prefetches next in the background. When a read crosses out of cur, next is promoted to cur and the following window is kicked off. On a sequential pattern the next window is already in memory, keeping the device queue full and hiding read latency behind compute. Correctness: writes and truncation cancel any in-flight prefetch and drop both windows (ext_ra_invalidate); the worker uses its own inode_ref (lwext4 locks its block cache internally); ~ext_vdata joins the worker and frees both buffers on vnode inactive. The tst-ext4-bench byte-for-byte verify passes (no VERIFY FAIL) for 64K/128K/256K reads.
The async double-buffered read-ahead only kept ~1-2 windows ahead of the consumer, so a single sequential stream drove the NVMe queue to depth ~1-2 and topped out ~1.25 GB/s vs Linux's ~2.3 GB/s (Linux issues many concurrent readahead requests, keeping the device queue deep). Replace the two-window (cur/next) scheme with a ring of RA_WINDOWS windows (256 KiB x 8 = 2 MiB per open file, same memory as before) filled by a pool of RA_WORKERS worker pthreads. Each worker calls the existing synchronous ext_internal_read() into a ring slot, so RA_WORKERS fills are in flight at once -> device queue depth ~RA_WORKERS. As soon as the consumer drains a window the ring slides forward and re-arms that slot for the window RA_WINDOWS ahead, keeping the pipeline full. Correctness: per-slot state (EMPTY/FILLING/READY) + a ring generation. All worker-shared fields are under ra_cvmtx; the read path holds ra_lock (outer) and takes ra_cvmtx to inspect slots. Workers never take ra_lock, so no deadlock. Seek/write/truncate bump the generation and drain in-flight fills before reusing buffers (no use-after-free); stale worker results whose generation no longer matches are discarded. Reads larger than a window or spanning two windows fall through to the direct read path. ~ext_vdata broadcasts shutdown, joins all workers, then frees the buffers. Reuses lwext4's existing bio path (ext_internal_read -> ext4_blocks_get_direct -> one bio + bio_wait) rather than issuing raw bios, which would mean re-implementing lwext4's extent->block mapping; a worker pool blocked in bio_wait maps directly to device queue depth and keeps all that code correct. Verified with tests/tst-ext4-bench.cc (absolute-offset byte verification) at bs 4K/64K/128K/256K/262143/1M and files smaller than the ring: no VERIFY FAIL.
Sweep on a local-NVMe host (, ext4 4K/no-journal, O_DIRECT so reads hit the device, 512 MiB, median of 3): config 64K 128K 256K prefetch mem/file N=1 M=1 ~0.69 ~0.85 ~0.85 256 KiB (~QD1) N=4 M=2 ~1.20 - ~1.28 1 MiB N=4 M=4 ~1.26 ~1.23 ~1.29 1 MiB <- knee N=8 M=4 ~1.28 ~1.28 ~1.31 2 MiB N=8 M=8 ~1.28 ~1.28 ~1.29 2 MiB N=16 M=16 ~1.25 - ~1.21 4 MiB Throughput climbs steeply from N=1/M=1 to the knee at N=4/M=4 (~1.28 GB/s) and then plateaus: deeper rings or more workers give no further gain. The cap is downstream of this code -- OSv's virtio-blk make_request() serialises submission under a single _lock and a single virtqueue, so effective device queue depth tops out at ~2-4 regardless of how many prefetch windows are in flight. (Linux fio O_DIRECT on the same raw NVMe: ~1.32 GB/s @128k QD1, ~1.68 GB/s @128k QD>=2, i.e. the device itself saturates at QD2; the remaining OSv gap is the guest virtio-blk path, not prefetch depth.) So N=4/M=4 is the sweet spot: same throughput as the deeper configs at half the memory (1 MiB vs 2 MiB per open file). It also matches the prior 2-window async design (~1.25 GB/s here) while using a general N-window ring. Make N and M build-time tunables (-DRA_WINDOWS_N / -DRA_WORKERS_M via $(RA_FLAGS)) so the sweep is reproducible when the virtio-blk submission path is later parallelised. No VERIFY FAIL at any config or block size (4K/64K/128K/256K/262143/1M, files smaller than the ring).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note
Stacked on #1431 (ext4 fsync + page-cache bridge). The two fsync/dtime
commits from #1431 appear at the base of this branch; once #1431 merges this
reduces to the four performance commits. Review #1431 first.
Closes most of the ext4 sequential-read throughput gap vs Linux, measured A/B on
a 2-socket x86-64 bare-metal host booting OSv under KVM against local NVMe
(no throughput cap).
The gap
Baseline ext4 (lwext4) vs Linux ext4 on the same NVMe:
ext_readissued one synchronous bio perread()with no prefetch(
ext4_blocks_get_directbypasses lwext4's block cache), so a stream of smallsequential reads left the NVMe queue idle during app compute. Linux keeps the
device busy via multi-window async read-ahead.
Changes (4 commits)
tst-ext4-bench): sequential write+fsync / read / 4K-random,MB/s, filesystem-agnostic. Writes an absolute-offset pattern and verifies
every byte on the read, so it doubles as a read-path correctness test.
uiomovecopy on the single-iovec fast path; also fixes a latentfree()/free_contiguous_aligned()mismatch. (Correctness cleanup — thecopy was not the bottleneck.)
window and serve subsequent in-window reads from a memcpy. +73% at 64K.
cur/next) + a per-vnodeprefetch worker that fills
nextwhile the app consumescur, keeping thedevice queue busy. A worker thread (not raw
bio_done) reusesext_internal_readverbatim; writes/truncation invalidate,~ext_vdatajoins the worker and frees.
Results (bare-metal host, local NVMe, ext4 4K/no-journal, median of 3, no VERIFY FAIL)
Reference on the same NVMe: Linux ext4 buffered read ~2300 MB/s; NVMe
single-stream O_DIRECT ceiling 1.5 GB/s @128k, 1.9 GB/s @1m. OSv's async ext4
read now exceeds the 128K O_DIRECT single-stream ceiling and sits ~within 2x
of Linux buffered. The remaining gap is queue depth — Linux prefetches many
windows, this prefetches one ahead (depth ~2); a deeper prefetch pipeline is the
upgrade path.
Write stays at Linux parity and random read ≥ Linux, so no change was needed
there.